Composition and templating with native handlebars#1954
Open
alexmojaki wants to merge 16 commits into
Open
Conversation
Managed variables can now reference other variables and render
Handlebars templates against typed inputs.
Composition: `@{variable_name}@` references in serialized variable
values are expanded during resolution, before deserialization. The
resolver walks a small Handlebars-compatible subset of `@{}@` block
helpers (top-level `#if`/`#each`) and supports dotted-path access
(`@{user.name}@`). Resolution priority is shared between top-level
variables and child references via `_BaseVariable._lookup_serialized`:
context override → provider → registered code default. Reference
cycles, missing references, and depth limits surface through
`ComposedReference.error` and propagate as a single composition
warning at resolution time, with the resolver falling back to the
code default. Each resolved variable also carries a `composed_from`
trail (with reason/label/version per ref) onto the span attributes.
Template rendering: a new `logfire.template_var()` registers a
`TemplateVariable[T, InputsT]` whose `.get(inputs)` resolves the
variable, expands `@{ref}@` references, renders any `{{placeholder}}`
expressions using `pydantic_handlebars`, and deserialises to the
declared type. `inputs_type` generates a `template_inputs_schema`
that is included in the variable config and synced to the server.
`pydantic-handlebars` is an optional dependency installed via the
`logfire[variables]` extra on Python 3.10+; calling `template_var()`
without it raises immediately rather than silently degrading.
`Variable` is now a thin subclass of an internal `_BaseVariable`
that holds the shared resolution pipeline. The base class carries
no template-related surface area; `TemplateVariable` overrides
`to_config` to attach `template_inputs_schema`, and external diff/
sync code gates on `isinstance(variable, TemplateVariable)` via the
`get_template_inputs_schema(variable)` helper.
Write-time validation lives in `logfire.variables.template_validation`
and uses `pydantic_handlebars.check_template_compatibility` to detect
undeclared `{{field}}` references across the composition graph. Cycle
detection on the reference graph is also exposed for push-time use.
- Adds `docs/reference/advanced/managed-variables/templates-and-composition.md`
covering Handlebars `{{placeholder}}` rendering via `logfire.template_var()`
and `@{variable_name}@` composition references, including structured
values, cycle detection, and how templates and composition combine.
- Expands the managed-variables index with a templates intro and links
to the new page.
- Mentions `template_inputs_schema` and the `[variables]` extra in the
configuration reference and nav.
- Adds `examples/python/variable_composition_demo.py` exercising
composition, structured variables, template inputs, and
composition-time conditionals end-to-end.
- Skips Python doc examples whose source mentions `logfire.template_var`
when `pydantic-handlebars` is unavailable (matches the runtime
requirement on Python 3.9).
Replaces the regex-and-translate workaround in reference_syntax.py and
composition.py with calls into pydantic-handlebars >= 0.2.0's native
configurable-delimiter API.
## What changed
- pyproject.toml: bump `pydantic-handlebars` to `>=0.2.0` (in both the
`[variables]` extra and the dev group) and exempt it from the
`exclude-newer` filter alongside the other Pydantic packages.
- `_handlebars.get_environment()` returns a cached
`HandlebarsEnvironment(open_delim='@{', close_delim='}@')` for the
composition pass. `extract_composition_dependencies()` wraps
`pydantic_handlebars.extract_dependencies` with the same delimiters.
- `reference_syntax.render_once()` shrinks to a one-line call into that
environment. The sentinel-protect-then-regex-translate code path is
gone; `{{...}}` runtime placeholders survive because they're plain
content under the configured delimiters.
- `composition.find_references()` / `_collect_ref_names()` now route
through `extract_composition_dependencies` for AST-correct detection,
then a textual position scan supplies first-occurrence ordering.
## What you can now write
`@{...}@` accepts the full Handlebars syntax — block helpers with
dotted-path headers, `{{#each}}` parent-context references (`../`),
helper subexpressions, the lot. Previously the regex translator only
covered top-level identifiers in block headers, so e.g.
`@{#if user.active}@` silently dropped its condition. New regression
tests cover these in `TestExpandReferencesNativeHandlebarsSyntax` and
`TestFindReferencesNativeHandlebarsSyntax`.
The unresolved-dotted-reference protection in `_render_value` is kept
for behaviour compat (literal `@{name.field}@` is retained in the
output when `name` can't be resolved). That's the
"misrender retention" parity question — addressed separately.
- Use exact, trimmed equality against 'null' in expand_references rather than a case-insensitive startswith match. JSON null is always lowercase, and 'nullify' shouldn't have matched. - Add a test that exercises the 'dotted ref whose base is resolved' branch in _protect_unresolved_dotted_refs, which the regex narrowing in this PR left uncovered.
…ble-composition-clean
…le-composition-native-handlebars
…tic/logfire into feature/variable-composition-native-handlebars
alexmojaki
commented
May 22, 2026
dmontagu
added a commit
that referenced
this pull request
May 24, 2026
Six items Alex flagged as important-but-not-blocking on the original 1951/1952 stack, folded in before #1954 merges: ## Docs - `composition.py` module docstring + `expand_references` docstring no longer claim block helpers are restricted to top-level identifiers — the native handlebars path accepts `@{#if user.active}@` and helper sub-expression headers. - `docs/.../templates-and-composition.md` Control Flow section rewritten to match: drops the "must be top-level" caveat and adds rows for dotted-path conditions and `../` parent-scope access from inside a block. ## Caching - New `compile_composition_template(source)` in `_handlebars` wraps the parse step in an `lru_cache(maxsize=1024)` (#1952 r3289095058). `reference_syntax.render_once` now compiles once per distinct source and reuses the compiled program across resolutions — managed-variable values are typically stable, so the hit rate should be high. - `extract_composition_dependencies` no longer re-runs the `try: from pydantic_handlebars import …` block on every call — the import is moved into a `@cache`d helper that returns the function object directly. Matches the pattern of the existing `_get_template_compatibility_checker` (#1952 r3288194502). ## Warning text - `_composition_failure` renamed to `_fallback_to_default` and takes a `failure_stage` argument (`'composition'` or `'template rendering'`) so the `RuntimeWarning` text reflects the actual failed step. A pydantic-handlebars parse error during `{{...}}` rendering used to surface as `"composition failed"` even though composition succeeded (codex finding). - Updated `test_remote_render_error_records_exception` accordingly. ## Test naming - `test_override_render_failure_falls_back` renamed its locals: `bad_override` → `templated_config` (it's a *valid* template), and introduced `invalid_inputs` for the inputs that actually fail the pattern constraint. Alex r3289312130. ## Escape-detection coupling - Added a comment on the `_HAS_REFERENCE` regex flagging that the lookbehind encodes pydantic-handlebars' current escape semantics ("any preceding `\` escapes") and noting it'll need to count preceding backslashes if pydantic-handlebars adopts Handlebars.js's odd-vs-even-backslash spec behaviour (#1952 r3289062247).
This was referenced May 25, 2026
alexmojaki
commented
May 25, 2026
Composition and render failures already warn before falling back to the code default, but a value that was fetched/composed/rendered successfully and then failed validation (including a context override that renders to an invalid value) fell back silently. Emit a RuntimeWarning in both paths so the substitution is observable, closing the asymmetry Alex flagged on #1954 (tests/test_variable_templates.py:549 / #1951 discussion r3289312130).
Contributor
There was a problem hiding this comment.
2 issues found across 3 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="logfire/variables/variable.py">
<violation number="1" location="logfire/variables/variable.py:363">
P2: Raw ValidationError text is emitted in warnings, which can expose input values.</violation>
<violation number="2" location="logfire/variables/variable.py:518">
P2: Validation fallback warning uses raw exception formatting and may leak validated input data.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| # outer handler falls back to the code default, so the silent substitution that | ||
| # composition/render failures already surface isn't swallowed for this path. | ||
| warnings.warn( | ||
| f"Variable '{self.name}' value failed validation; falling back to code default: {e}", |
Contributor
There was a problem hiding this comment.
P2: Raw ValidationError text is emitted in warnings, which can expose input values.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At logfire/variables/variable.py, line 363:
<comment>Raw ValidationError text is emitted in warnings, which can expose input values.</comment>
<file context>
@@ -353,7 +353,18 @@ def _resolve_context_override(
+ # outer handler falls back to the code default, so the silent substitution that
+ # composition/render failures already surface isn't swallowed for this path.
+ warnings.warn(
+ f"Variable '{self.name}' value failed validation; falling back to code default: {e}",
+ category=RuntimeWarning,
+ stacklevel=2,
</file context>
Suggested change
| f"Variable '{self.name}' value failed validation; falling back to code default: {e}", | |
| f"Variable '{self.name}' value failed validation; falling back to code default: {e.errors(include_input=False, include_url=False) if isinstance(e, ValidationError) else e}", |
| # back to the code default, mirroring the composition/render failure warnings so the | ||
| # substitution isn't silent (the asymmetry Alex flagged on #1954). | ||
| warnings.warn( | ||
| f"Variable '{self.name}' value failed validation; falling back to code default: {value_or_exc}", |
Contributor
There was a problem hiding this comment.
P2: Validation fallback warning uses raw exception formatting and may leak validated input data.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At logfire/variables/variable.py, line 518:
<comment>Validation fallback warning uses raw exception formatting and may leak validated input data.</comment>
<file context>
@@ -500,6 +511,14 @@ def resolve_ref(
+ # back to the code default, mirroring the composition/render failure warnings so the
+ # substitution isn't silent (the asymmetry Alex flagged on #1954).
+ warnings.warn(
+ f"Variable '{self.name}' value failed validation; falling back to code default: {value_or_exc}",
+ category=RuntimeWarning,
+ stacklevel=2,
</file context>
Suggested change
| f"Variable '{self.name}' value failed validation; falling back to code default: {value_or_exc}", | |
| f"Variable '{self.name}' value failed validation; falling back to code default: {value_or_exc.errors(include_input=False, include_url=False) if isinstance(value_or_exc, ValidationError) else value_or_exc}", |
The platform rejects user-created labels named 'latest' or 'code_default', so the SDK can treat them as unambiguous wherever it special-cases them (follow_ref and push-time template validation keyed by label name).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#1951 combined with #1952, i.e. #1952 directly against main, since I think that makes for cleaner review instead of the stack.